Hardware and software testing of networked devices is an integral part of the development and deployment processes. Testing is required to ensure consistent and predictable operation of the networked device under various operational conditions and to ensure the device will integrate successfully within networks. This is particularly important in networks having a wide range of networking devices produced by a variety of vendors such as telecommunications equipment. Test automation environments or testbeds are developed to automate the testing and validation process to rigorously exercise device functionality and ensure compliance with design specifications.
In the testing of Internet Protocol(IP)-addressable networked devices such as telecommunications networking equipment a Subject Matter Expert (SME) creates a test plan detailing how the networked device or hardware should be tested in order to ensure the hardware provides the desired design functionality. The test plan is converted by a programmer to code such as a test script which ideally implements the test objectives defined by the SME. The test script comprises commands for executing queries and configuration changes against the target hardware to achieve the desired objectives. The test plan will also detail the network configuration required to simulate real network conditions.
In order to speed up the development process, individual telecom equipment manufacturers have developed automated test environments dedicated to testing specific types or brands of hardware to implement the test plans. The automated test environments which may be implemented on a general purpose computer or by dedicated test hardware may be either directly connected to the target hardware or connected by a communications network to the target hardware. The automated test system may also provide simulated traffic or control third party traffic generators to simulate desired network conditions. Most equipment providers design, implement, support and maintain their own test automation environments using widely available programming languages such as TCL (Tool Command Language), PERL (Practical Extraction and Report Language) and Expect. The test scripts created in these test automation environments use text based commands for interaction with the devices under test through a command line interface system. The command line interface may be implemented over a network terminal protocol such as Telnet, SSH (Secure shell) or other direct connection communication such as serial based communication protocols. Alternative protocols may also be used such as simple network management protocol (SNMP), TL1, XML, and others for querying and controlling devices.
In more complex testing environments involving multiple brands of hardware, and possibly different software loads, the complexity of the automated testing environment grows significantly. Understanding and developing unique test scripts for individual vendor equipment can be laborious and time consuming and provide inaccurate results if the test scripts are not adapted properly. Managing a testing environment can become even more complicated when trying to integrate multiple test platforms required to test a series of devices on a network. In order to effectively test networked devices or nodes the programmer has to ensure that the test script achieves the desired testing objectives defined by the SME. This can be particularly difficult if the programmer is limited in understanding the operation of the hardware and software involved in the testing process. Another complicating factor in test automation is the accelerated development of new technologies having new and unique hardware features and configurations which will either require a unique testing system or only provide limited ability to be tested using current automated testing environments.
Accordingly, systems and methods that enable flexible automated testing of diverse networked devices remains highly desirable.